Skip to content

stop "uncompressing" registry data #4470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 4, 2025
Merged

stop "uncompressing" registry data #4470

merged 2 commits into from
Nov 4, 2025

Conversation

@KristofferC
Copy link
Member

@KristofferC KristofferC commented Oct 26, 2025 •
edited
Loading

instead, do all operations on top of compressed data (VersionRange -> data) instead of uncompressed (VersionNumber -> data).

This avoids having to materialize dictionaries with compat data for every version.

As an example:

Pkg.activate(; temp=true)
Pkg.add("Plots")
 _, _, reg_instance = first(Pkg.Registry.REGISTRY_CACHE)[end]

# master
julia> (Base.summarysize(reg_instance) - Base.summarysize(reg_instance.in_memory_registry)) / 1024^2
29.9 # MB

# master
julia> (Base.summarysize(reg_instance) - Base.summarysize(reg_instance.in_memory_registry)) / 1024^2
66.6 # MB

(where I am removing the size of storing the raw data of the registry itself).

and


julia> Pkg.rm(["Plots"]); empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add(["Plots"])

# PR
  1.320097 seconds (8.84 M allocations: 587.032 MiB, 11.50% gc time)

# master
  1.397698 seconds (9.65 M allocations: 718.492 MiB, 13.56% gc time)

@github-project-automation github-project-automation bot moved this to New in Pkg.jl Oct 26, 2025
@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch 5 times, most recently from e190b0a to 3e23b37 Compare October 28, 2025 18:33
@KristofferC
Copy link
Member Author

KristofferC commented Oct 28, 2025

Test seems to pass now and updated figures for

try Pkg.rm(["Plots"]; io=devnull) catch end; empty!(Pkg.Registry.REGISTRY_CACHE); @time Pkg.add(["Plots"]; io=devnull)

are

# master
  1.117075 seconds (8.07 M allocations: 644.749 MiB, 1 lock conflict)

# PR
  0.800553 seconds (6.86 M allocations: 466.209 MiB)

and specifically the deps_graph step where we gather information from the registry is

#master
  collect deps
  0.129429 seconds (2.61 M allocations: 290.017 MiB)

# PR
  collect deps
  0.063639 seconds (1.43 M allocations: 115.601 MiB)

I still need to take a proper pass through this because there is honestly quite a bit of AI slop in here, but I just wanted a finished implementation so I could look at the numbers and see it was what I thought it would be.

@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch from 290a468 to 3cfefbc Compare October 28, 2025 19:23
KristofferC and others added 2 commits November 4, 2025 12:56
@KristofferC
stop "uncompressing" registry data
78a03de 
instead, do all operations on top of compressed data (`VersionRange` -> data) instead of uncompressed (`VersionNumber` -> data).

This avoids having to materialize dictionaries with compat data for every version.
some more optimizations
05c6e20
@KristofferC KristofferC force-pushed the pkg-registry-no-incompress branch from 3cfefbc to 05c6e20 Compare November 4, 2025 13:16
@KristofferC KristofferC marked this pull request as ready for review November 4, 2025 13:16
@KristofferC KristofferC merged commit efe1eaf into master Nov 4, 2025
11 checks passed
@KristofferC KristofferC deleted the pkg-registry-no-incompress branch November 4, 2025 14:15
@github-project-automation github-project-automation bot moved this from New to Done in Pkg.jl Nov 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backport 1.13

Projects

Pkg.jl
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@KristofferC